1. Field of the Invention
This application relates to the field of digital data coding and compression and more particularly to the field of coding digital data for video conferencing in a manner that reduces the amount of data provided in response to errors in received data.
2. Description of Related Art
A video signal includes data representing a temporal succession of still images which are referred to as xe2x80x9cframesxe2x80x9d. If the time between each frame is small enough and the frames are displayed in temporal sequence, the video signal simulate motion. For example, the NTSC television standard provides for transmitting a video signal with thirty frames per second. On a conventional U.S. television set which receives NTSC video signals, the display is thus updated thirty times per second to provide a simulated moving image.
Transmitting video signals over data channels having relatively low band width provides challenges. Generally, there exists a tradeoff between the number of frames per second which may be provided and the resolution of each frame. Increasing the resolution of each frame increases the amount of data corresponding to each frame. Thus, for relatively high resolution frames, the number of frames per second which may be transmitted in a low band width channel is smaller than the number of frames which may be transmitted per second in connection with low resolution frames.
One solution for improving the performance of video signals transmitted over relatively low bandwidth channels is to eliminate the need to transmit each frame completely. Instead, what is transmitted is other information that allows reconstruction of each frame at the receiving end. For example, the H.263 ITU standard provides for transmitting a first frame in its entirety and then transmitting one or more motion vectors and prediction error information for subsequent frames. The motion vector(s) represent translation of portion(s) of the image from a reference frame to a current frame while the prediction error information represents the difference between the current frame and the reference frame with the motion vector information applied thereto. Thus, each video frame may be determined by applying the motion compensation vector to the reference frame and then applying the prediction error information. The video signal can then be transmitted by sending the first frame followed by motion vectors and prediction error information for each of the subsequent frames.
The H.263 standard also allows for relative coding of a portion of a frame, which is called a xe2x80x9cmacroblock.xe2x80x9d In the case of using a macroblock, the vector and error information may be provided relative to the macroblock, as opposed to an entire frame. In addition, instead of transmitting an entire new reference frame, it may only be necessary to transmit a new macroblock, which may then be used for subsequent relative encoding with motion vector(s), error compensation, etc.
A drawback to the H.263 technique for encoding video information is that, since each macroblock is based on the motion vectors and prediction error information applied to the previous macroblock, then a transmission error or data loss causes all subsequent macroblocks to be suspect. Recovery from such an error may be performed by the transmitter resending a new macroblock. For transmission channels with a relatively low error rate, this does not pose a significant problem. An enhanced version of the H.263 standard, deemed H.263+ or H.263 version 2, provides a mechanism to indicate to the transmitter that the received signal contains errors. However, the H.263+ standard has limitations, some of which are described in annex N.5 of the H.263+ standard specification.
Thus, when the receiver receives motion vector information and prediction error information that is incorrect, the receiver sends a signal back to the transmitter to request that an entire reference frame be sent so that error recovery is possible.
According to the present invention, transmitting and receiving video data includes an encoder storing macroblocks in a plurality of reference buffers, the encoder transmitting macroblocks that are encoded relative to each of the reference buffers, a decoder receiving the macroblocks, and in response to the decoder indicating an error with respect to a particular one of the macroblocks encoded relative to a first one of the buffers, the encoder using a second one of the buffers for error recovery if there is no error associated with the second buffer. In response to the decoder indicating an error with respect to a particular one of the macroblocks and all of the buffers having errors associated therewith, the encoder may transmit at least one macroblock that is encoded independent of any other macroblocks. The decoder may receive a macroblock having an error associated therewith and, in response to all buffers of the decoder having errors associated therewith, the decoder may transmit a request to the encoder to transmit at least one macroblock that is encoded independent of any other macroblocks. In response to the request, the encoder may transmit at least one macroblock that is encoded independent of any other macroblocks.
The encoder using a second one of the buffers for error recovery may include coding a macroblock using a second one of the buffers as a reference and marking the first one of the buffers as having an error. In response to marking the first one of the buffers as having an error, a timer may be set to indicate an amount of time since the error occurred and, in response to the amount of time since the error occurred being greater than a predetermined threshold, the error associated with the first one of the buffers may be cleared. The predetermined threshold may correspond to a round trip transmission and processing time between the encoder and the decoder.
A number of buffers used may be increased in response to errors requiring the encoder to transmit at least one macroblock that is encoded independent of any other macroblocks and decreased if, after a predetermined amount of time, no errors requiring the encoder to transmit at least one macroblock that is encoded independent of any other macroblocks occur. Each macroblock may be associated with a temporal reference and a table may be provided that maps each temporal reference number with one of: a particular one of the buffers and a value indicating that a macroblock corresponding to a temporal reference number is not being stored.
According further to the present invention, transmitting video data includes an encoder storing macroblocks in a plurality of reference buffers, the encoder transmitting macroblocks that are encoded relative to each of the reference buffers, and, in response to an error with respect to a macroblock corresponding to a first one of the buffers, the encoder using a second one of the buffers for error recovery if there is no error associated with the second buffer. In response to an error with respect to a macroblock and all of the buffers having errors associated therewith, the encoder may transmit at least one macroblock that is encoded independent of any other macroblocks. In response to a receiving a request, the encoder may transmit at least one macroblock that is encoded independent of any other macroblocks. The encoder using a second one of the buffers for error recovery may includes coding a macroblock using a second one of the buffers as a reference and marking the first one of the buffers as having an error. In response to marking the first one of the buffers as having an error, a timer may be set to indicate an amount of time since the error occurred and, in response to the amount of time since the error occurred being greater than a predetermined threshold, the error associated with the first one of the buffers may be cleared. The predetermined threshold may correspond to a round trip transmission and processing time between the encoder and the decoder. number of buffers used may be increased in response to errors requiring the encoder to transmit at least one macroblock that is encoded independent of any other macroblocks and may be decreased if, after a predetermined amount of time, no errors requiring the encoder to transmit at least one macroblock that is encoded independent of any other macroblocks occur. Each macroblock may be associated with a temporal reference number and a table may be provided that maps each temporal reference number with a particular one of the buffers or a value indicating that a macroblock corresponding to a temporal reference number is not being stored.
According further to the present invention, receiving video data includes a decoder receiving a plurality of macroblocks that are coded relative to a plurality of buffers corresponding to a plurality of macroblocks and, in response to an error with respect to a particular macroblock, the decoder marking for error a first one of the buffers associated with the particular macroblock. Following marking for error a first one of the buffers, the decoder may receive a macroblock coded relative to a second one of the buffers. Following marking for error the first buffer, and in response to all buffers of the decoder having errors associated therewith, the decoder may request at least one macroblock that is encoded independent of any other macroblocks. Following requesting at least one macroblock that is encoded independent of any other macroblocks, the decoder may reset a timer and may forgo requesting an other macroblock that is encoded independent of any other macroblocks until the timer has exceeded a predetermined threshold.
according further to the present invention, an apparatus that transmits video data includes a plurality of reference buffers that store a plurality of macroblocks, a reference processor unit, coupled to the buffers, that provides at least one motion compensation vector and a motion compensated image of a macroblock provided thereto, a difference unit, coupled to an input video signal and to an output of the reference processor unit to provide a difference therebetween, and a coding controller that controls transmission of video data to provide each macroblock as either a macroblock that is encoded independent of any other macroblocks or a macroblock coded relative to an output provided by said difference unit where, in response to an error for a macroblock corresponding to a first one of the buffers, the controller causes the reference processor unit to use a second one of the buffers for error recovery if there is no error associated with the second buffer. In response to all of the buffers having errors associated therewith, the controller may cause transmission of at least one macroblock that is encoded independent of any other macroblocks. In response to receiving a request, the controller may cause transmission of at least one macroblock that is encoded independent of any other macroblocks. A number of buffers used may be increased in response to errors and may be decreased in response to not receiving errors after a predetermined amount of time.
According further to the present invention, a computer storage medium containing a computer program includes means for storing macroblocks in a plurality of reference buffers, means for transmitting macroblocks that are encoded relative to each of the reference buffers, and means for using a second one of the buffers for error recovery if there is no error associated with the second buffer in response to an error with respect to a particular macroblock corresponding to a first one of the buffers. The computer storage medium may also include means for transmitting at least one macroblock that is encoded independent of any other macroblocks in response to an error with respect to a particular macroblock and all of the buffers having errors associated therewith. The computer storage medium may also include means for detecting a request for at least one macroblock that is encoded independent of any other macroblocks.
Means for using a second one of the buffers for error recovery may include means for coding a macroblock using a second one of the buffers as a reference and means for marking the first one of the buffers as having an error. The computer storage medium may also include means for setting a timer to indicate an amount of time since the error occurred in response to marking the first one of the buffers as having an error and means for clearing the error associated with the first one of the buffers in response to the amount of time since the error occurred being greater than a predetermined threshold. The predetermined threshold may correspond to a round trip transmission and processing time.
According further to the present invention, a computer storage medium containing a program for receiving video data includes means for receiving a plurality of macroblocks that are coded relative to a plurality of buffers corresponding to a plurality of macroblocks and means for requesting a particular macroblock be resent and marking for error a first one of the buffers associated with the particular macroblock in response to an error with respect to the particular macroblock. The computer storage medium may also include means for receiving the macroblock coded relative to a second one of the buffers following requesting the particular macroblock.
According further to the present invention, transmitting video data includes an encoder storing macroblocks in a plurality of reference buffers where each macroblock is associated with a temporal reference number, providing a table that maps each temporal reference number with a particular one of the buffers, wherein for a subset of the temporal reference numbers, a value indicates that none of the buffers correspond thereto, the encoder transmitting macroblocks that are encoded relative to each of the reference buffers, and in response to receiving an error message indicating an error with respect to a particular one of the macroblocks encoded relative to a first one of the buffers, the encoder using the table and a second one of the buffers for error recovery if there is no error associated with the second buffer. For each buffer, a data structure may be provided containing values for a corresponding one of the temporal reference numbers, an indication that an error has occurred, an indication that an error requires recovery, an indication that the buffer has been used for recovery of another thread, and a timer indicating an amount of time since recovery of the buffer was undertaken. In response to receiving an error message indicating an error with respect to a particular one of the macroblocks encoded relative to a first one of the buffers, values for the first buffer indicating that an error has occurred and that an error requires recovery may be set to TRUE and the timer indicating an amount of time since recovery of the buffer was undertaken may be set to zero. After using the table and a second one of the buffers for error recovery, the value indicating that an error requires recovery may be set to FALSE. After setting the value indicating that an error requires recovery to FALSE, the timer may be incremented according to wall clock time. In response to the timer being greater than a predetermined value, the value indicating that an has occurred may be set to FALSE
The system described herein provides a mechanism for using multiple reference buffers in a manner that reduces the tendency to transmit entire frames in instances where the transmission channel has a relatively high bit error ratio. In addition, the system also reduces the amount of time a received video signal may be frozen waiting for error recovery in the form of an entire frame.